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Thlii documont pror>ontG tl\o design ot an evaluation 
study tor the Proqratn ot Comprohonsivo Grants for Teacher Education 
which i tundod by tho National Science Foundation The main 

concf^rns ot th'i^ evaluat ion ore the need to provide context 
intormation to i ndi vid ua 1 -pr oiec t dir ectors/* and to develop a 
summative strategy for tho national proqr'ara. A total ot 17 
. . i n: trumt r. ts were seli^cted to cotleot a variety of cognitive and 
af.ft^ct*ivc data for both science and raa themat ics projects from 
students, teachi^rs and administrators. Approximately one-half ot 
thtsr i nst r ument'7^ were developed for this evaluation while the 
r-'ciainder were chosen from existing instruments. -A total of 35U 
:;chcols froE five ^^xper iner.t al regions and five control regions 
partici tod in this s^udy. The data qatherod by this national design 
Will b»- processed and aValyzed during tho next several months\and 
pr^^sented to Com pr^he n siVe Project Directors and to the NSF. (HM) 

. . . 
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A SHIATKGY lOR KVALUATIKG TllK NSF COMrRKlIKNSl Vi: rUOCRAM FOR TFACHKR KDUCAT lOi^I 

Wayne W. Welch and Arlen K. (Ui 1 1 i ckr.on"^ 
University oi Minner.ota 

Among the more innovative of recent NSK prC)j;ram clianges is a^ venture entitled 

the Progt^am of Coifij^rolio'ns i ve Grants for Teaclier Kducation. Tlie shortened label 

for this program is "Compreliens ives , " Altliougli tlie program has be^n in exis-xence 

lous than two years, it is g^^uning tlie status and attention of dtlier well-known 
• * • .• ' . * 

NSF teacher education programs, sucli as tliQ Summer and Academic Year Institutes, 



and the Cooperative College-Sclvool Science (CCSS) program. 
The goals of the Compreliens ive Program are twofold: 



1. To help schooAs ,' thorough tlie education of tlieir instructional, 
resoj.ir.ce and /sujTervisory personnel, in developing tlieir capacity 
for se 1 f- improvement in science and mathematics education, and 

/ 

2. To assist the efforts of colleges and universities in developing as 
part ,of.> their regular activities m.ore effective programs for 'tlie pro- 
.service" and i.n-service education of science and mathematics teachers. 

Five experimental projects were funded under this program in 1972. The 

projo|:ts wore fundc(^' fcfr four years. Mathematics projects were estabTished at 

- Notre Dame and San-Jose. Science Comprehens ives were located in Mississippi, 

South Dakota', and Wyoming.' Tliese five projects are tlie focus for the evaluatit)n 

strategy descrit^cd in this paper. 

The unique Ifeatures of the Program that should be mentioned here include: 

(1) a regional rather thart national focus, (2) active responsiveness to school 

needs, "(3) program flexibility, and (4) multi-year funding. A fifth distinguishing 

characteristic of the Comprehensive Program is its attention' to ongoing planning 

and evaluation. It is this last point that serves as the substance of this paper. 

What' str^ategy would be appropriate for evaluating the impact and efficacy of the 

Compreher^ ive Program? , • j , 

jThe authors are "^indebted -to Alehouse Buccino, Glenn P-racht, James R\it her f or d 

a ^ ' - ■ 

i-nf^" and Herbert Walberg for con:ment:s on an earlier draft of this article. 



The u,ulorlyi„, con..: id. rat: Ion 1 n •co.uluc n, t hi. . ov .ny other. ovnU.aUon 
1^ thnt it. function is ,o r.. tlu>y n fonnat i on for Cho purpose of ,naku\, dec- i . i on-s 
(Welch. 1969). information pyrtainin, Co two pri.nary decision siCunt:ions was 
-ur.hf i„ the present study: (l/ reiio,u.l needs .>,.sessnK-n t data to He used b, 
project directors iu developi|,c the various co.aponents- of their ccnpreh^ns ive ' 
project, and (2) pretesfdaCa ..ninst'which^ potent ^^.'l posttost ch.n.es couU,' be 



detected. 



Thcmnin concerns of the evaluation were the need "to 'prov^^ntext ' 
information- to inJiyuU.al project directors and 'to develop a su^.a^^rategy ' 
for the national program. ^ evaluation strategy to address these problo.s is 
described below. ^ - ^ 

■I 

J Evaluation Plan 

_ To' obtain data bearing on the severa 1 . ob j ec t i ves of the program, a multi- 
level tiosting^program'was implemented. A variety of cognitive and affective 
data was .sought for both science a.'.d mathematics projects from students, teachers, 
and administrators. ;^he final selection of instruments was based upon the under/ 
lying Philosophy of th^ Comprehensive Program (Rutherford. 1971). availability of 
instruments, arrd- Tr^vice^of an external Advisory Con^ittee. (The latter grou^ . 
consisted of six members selected from the areas of science and' mathemat ics ^ 
education, educational Ip.sychology. school systems, and the.^F.) 

Fro« more that three dozen instruments cons idered". -a total of 17 were 
selected. Approximately one-half were developed for this, evaluation while the; 
remainder were chosen from existing instruments. These instruments are listed" ^ 
in Table 1 together with the appropriate region. 



^ 
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I Insert Tabic 1 about here 



A brief ^dcGcripti,<3'n of' each instruniont foil 
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Evalu^itlon Instruments 





; Principals 




Teachers • ' , . 
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1 ' Students - - 
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Principal Questionnaire 
Work Values Inventory 
Mathematics Inventory. ^ 


• ' . -A- 

National Teachers Exam-Mathematics 
^)fet:hematics Inventory for Teachers 
Teacher Questioiinaire " 


Mathematics Achievement Test 
Mathematics At^titude Inventor 
Learning Environment Iiwentor 


Wyor.iing, 
pi, South 
d their 
egions) 


^ Principal Questionnaire 
Science Process Invei^ory 
Science Attitude Inventory 


National Teachers Exam-Science 
Science Process Inventory 
ScientG Att;itude Inventory 
Xe^cher Questionnaire 

\ 


Test of Achievement in Scienc 
Science Process Inventory 
Science Attitude Inventory 
Learning Environment Inventor 



ven to control sample ' > , . ' 
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Test of Achievement in Salience (NAKP, 1970) 3, 

• 

A. Form I • , ^ 

A 'A5-itein multiple Choice t:or.t for .secondary school science comprised 
of items selected from tlie National Assessment Test for Science. 
Items were selected witli difficulty levels between' 57^ and '907o from 
those released in 1971 for 17 year-olds. 

'-^^ . * . 

B. Fonn II 

A ^*0-item multiple choice test comprised of items selected - from > the 
Natioiuil Assessment Test for Science. Items on this test were drawn 
from those developed for 13 year- olds . Tlie test was used in 8th 
grade science classes. 

f 

Science Process Inventory (W-elch 6c Pella, 1967) 

This instrument consists of 135 agree-d isa^^ree statements that describe 
the assimipt ions , activities-, products and ethics of sc ience . The 
reliability is .86. 

Science Attitude Inventory (Moore & Sutman, 1970) 

A Likert-type attitude measure designed to measure rlnte 1 lectual and 
emotional scien,tific attitudes. - The 60-item test has a reported >test- 
rctest reliability of .93. 

Learning Environment Inventory (Anderson , 1971) 

A 70-item measure of the students* perception of the secial/emotional 
atmosphere in the class.. Consists of 10-seven item scales labeled: 
Diversity, Formality, Friction,- Goal-Dire^:tion, Favoritism, Difficulty, 
Democratic, Cliqueness, Satisfaction, and Qisorganization . 

Mathematics Inv6ntory fbr Teachers 

The iHIT is a 30-item ins'trument comprised in part of items from attitude 
items in the NLS>iA Report (196^), and the International Study of 
Achievement in Mathematics (1967). The five Likert scales are designed 
to measure teachers*: Attitude toward mathematics as a process, attitiide 
toward classroom management practices, interest in teaching mathemat icS , • 
rating of teaching practices, and rating of teacher concern for students. 

Mathemat ics Attitude Inventory 

The >tAI consists of six eight-item Likert scales and is designe^^ to 
* measure the secondary' student ' s perception of: The math teacher, the 
value of mathematics in society, his enjoyment of niathematics , his anxiety 
toward mathematics, his self-concept in mathematics, and' his motivation 
in mathematics. Pilot study reliabilities for thQ six scales range from. 
.66 to .87 with four of the six having reliabilities greater than .80. 

Teacher Questionnaire 

Two questionnaires were developed, one of 121 items for mathematics 
teachers and one of 137 items for science teachers . They were designed 
to obtain a broad view of the school's mat^ema/iics and'science programs.. - 
The- quest ionnaires focused on the teac!-ier*s perception ofjiis personal 
background, skills af^ abilities, act ivit ies , and teaching environment.* 

. y 7 -- * ' ' 




Principal Qucntionnaircs/ 

T\.'o 90-il:oin quostionnairon vcvo developed, one for ^principals of 
nathenaticG t'.e*:cheri; , and one for principal:; of r.cience tcacMiers. 
The questionnaires focuiied on the ::chool and its respective science 
or nvithci-natics denartuent \;ith an enphasis on pro^,rans and activltj-os 
that mi[;ht be carried, out throii[;;h the Conprehensivc ProjccUs. A 
tiunber of items paralled teacl^er questionnaire items in order 'that ^ 
teacher and principals perceptions could be compared. 

Mathc^matits Achievement Test (nLSM\,lCG8) 

A. Foru I (Grade 3)_^ 

This test is comprised --of-niultiplc choice items selected 
from a pool of clchth j^radc Rational Lonsitudinal Study 
of Mathematics Achievement (IILS:L\) items. The selected items 
have correct i^tsponsc rates of bc'twccn .50 and .^0 and point 
•biserial r's of at least 0.3. 

B. Forn II (Grade 11) 

A AO-it^m, multiple choice test comprised of items selected from 
a pool of eleventh, sradc ,iILSlL\ items.. The selection criteria 
were identical to those for the Sth ^radc test. 

National Teachers E::aa (Educational Testing Services , 1968) 

A, *Mathcnatics • * 

The test consists of ^ 120 multiple choifc^ items, 10-15 per cent 
of V7hich measure a teacher's k^OT/ledge bf teaching secopdar^' 
* school mathematics. The remaining items provide a ir^asure of the 
content of secondary school mathematics. The test has a KR20 
reliability of .94, and offers normative data against uhich" 
teachers of an imoact region can be described. 

J . 

Biology and General Science 

A measure of knowledge about teaching biology and general science 
and the content in biology and general science at the secondary ' 
school level. The test consists of 120 multiple choice items, 
has a KPv2o reliability of .92 and offers normative data against, 
n;hich teachers of an imp^ict region can be described. • 

Chemistry-Physics and General Science ' 

This test measures I'.nouledge of teaching chcmistr>', physics, and 
general science and the content of the three subjects at the 
secondary' school level. The test consists of 120 multiple choice 
items, and has a III^^q reliability of V92^^ 

. TJorhy^alucs Inventoi-y (Super, 1970) ^ 
i • 

part^f the research effort of the project is focused on, the values 
and a'ttitddcs of secondary school administrators. This instrument . 
provides clata on 15-scales rr^lated to ^7ork values. The reliability 
of the scales range from .74 to .Go v;ith a mcclian of^ ;S3. 
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/ ^ Coiitx;ol Snmplii 

Because, of tho need i<o oxomiue cluu\|>^s on' n given iniivic f rogi on during the 
four years of the Projoctr., appropriate control regions were identified.... This ( 
proce^Vys and its rationale are de.scribed in detail in Gul 1 ickson , and Welch ( 1972). 
.The control, regions are basically tlie regions or states adjacent to the ex[)eri- 
mental regions. For example, the control region for"" Missi ss ippi is Alabama; for 
San Joise it is Southern California, and so on. These regionS.wcre selected in 
order to reduce the variance 'due to regional differences across the country. The 
number of schools sampled within each^ coritrol region is appr oxihiate ly one-half 
the number included^An the experimental region. This, decision\was based on cost 
factors alone. ^ t. 

figure 1 shows a map identifying the five Comprehensive Projects and their 
corresponding control regions. 



Insert Figure 1 about here; 

_ 

Testing Plan ^ 
Although the development and selection of the appropriate instruments 
represented major tasks, there was the additional problem of sampling the schools 
that would be involved in the study, T\\e major considerations in this selection 
process are described elsewhere (GuHickson & Welch, 1972) and only will be 
summarized here. The primary consideration was the number required to detect at 
the p<.05 revel a difference of one-half standard deviation. Table 2 presents 
the expt^cted number of participants from each region and within each category. 
The states included in each experimental pnd control region are also listed. 



Insert Table 2 about here 



FIGURE 1 ; 
PROJECT npACT REGIONS 




3JECT REGION'' 



\TROL REGION 



NOTE: The Twin Cities were not included in the South Dakota control. 
Birmingham was not included in the Mississippi control region. 
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P.fcaui,e of anticip.L.J .'-.oM-ro.-in-:i.!rrU r; , v.y. cat- s.-.;:,p lod in onch rcj^ion by 

i 

1^.0.::. A k.:y cpn::id.;rario:i in U..; r:.-::v.p 1 i .,;,> pKm to make certain that • Sufficient 

ran-.her:;. of re ;. p> !, v.'.to ■'.■a cicli cell to {.Tovicic valid nvctls a *;;e.s :'i^:xMit data 
to i.rojcct ilircctorr. . 

""• '■-('■nt of project ir. re vc,;i le<i .by c on:ri d er i n;; that the 

total .^vnluat iu;> pr-.v i.!--. for I . ■ t i ...-arly cOi) principal;;, an equal nunib.'r of ' 
tea.-h-rr-,, a„d than ?1,0';0 .•; t ndo r, L . In .:<Jdition. data were gathered on 17 

dificr. lit in.-.tru-..-nf; f r. ..i ncl-.oc.l :;catter.:d -Jirc/nJJhoi^t 15 .'.tater,. 

Ilcca.i-:e (;r fl,.- iv;(,fold ,:oai of 1. 1, i r; rvaliiatibn ;;tudy, i.e., regional needs 
r--.-;. ;:.,.nt data and protr'.L <!ata, we d<Ti,hd to l.old n r.erier; of ref;ional n.ectincs 
l.Tinr.iM;: to;..,-th.r t ...<■!„■ r . pr i i;e i pa I r. , projrct d i r . c f; , , r , and :;SF r.'[/....;entat iv.'s . 
hourl.-.M. .•a,<d. 1 .Ml i:,..,-; ,.or.- in thr .'Iprin;; o! 107?. 71, o t i np, provided 

• >'•:: i ' V ■ ■ V ,.( ; . V : I ; !■ . . I';:'.;, i,(h;catioa prop,ra;:r: -.ve'rc 

<-xplair;..d .5.'f..il f -Juun] p r •, o, a 1 ; aidly, O :i;j,reh.-nr; i ve Project 

Dir.etor-. w.r.- p!-,,vi.hd with .ai oppor t na i t y to iiite'ract wiih .■.rir,,o] people in 
tlicir p-ct i v.- i-:|...rr r^-;: i ni, ; .snd t.liijvily, v;i 1 ti.ib 1 ti'::i dat;! wrrc olW.nin^-J 
frrr::i f.-.MlMT:, .ir.'I i>i-iur i{,.tl Jn ndfiit.irui, wr wcru* ablv- r(; i)rovi(lo U^nchd::; wit:!i 

'•^•^'■^-t -'J -It. H-it.! l/.',.- t rr-M Ir...;:. Mm::., c ] ;i r; . ■ u:^lul: a'';y:;roin of 
riiiiHo::. roll,, t inn will, in rl.i-. (],;,f i nr m- ;t r ••; flir' iuji.L.t of iiu- a -wir r-J 

t>nt n [i ij f.-.ti,,^; fir,,, inr .'my n,,r ( 1 1'l^ ■ r 1 1. (W-iIlu-r,; ^l■/r](•|l, 1907). Dat.i 

wrrr col ) r.<-t i-fl -.K ,,J| .,n.OM-:Iy lifr;, i;,n'lM:M I y •;,'], -cMrl tlilrWs M til.- r.l/i-^r; du tlircr 

'■'i^'^/^'^" in-.t r r,! '.. T) n ' y t . ..j.,... t 1 . r r r- rl i ] fr • r r • n f. tr-,?,-;, 

/ 

I>nt J'r <jii i f r only A'; tiiftiufr' rj;,-,,', p('rifj»|. 



Duo to financial limitations, wo decided early in the process to gatlicr our 

control sample data tliro,iich tlic mail. Tliis meant forccoing acliicvemcnt testing of 

■ \ ^ ■ 

tcachcrr>, but it did provide n relatively inexpensive way to gatlior the remaining 

datn; 'As mentioned earlier (Table 2), 262 schools comprised the expected control 

Kroup. Here al.so, we ovcrsamplcd by AOZ because of anticipated non-'response . .We 

received c^oniplctod tests from 230 control sch(^*ls, approkimalely 857. of the 

expected total. A;>ain, it is our judcmont tliat this number is adequate for 

comparison purpos.es. 



Discussron 

1 



In this as in other evaluation stratc.^ies, the success of the plan is \deter- 
nined by the us.oful'ness of the i n f or::ia t i on to the several decision makers. Thc^'^ 
data C'Hiiered by this national design will be procd'Ssed and analysed during 'the 
next several' months and presented to Comprehensive Project Directors and to the 
National Science Foundation. ^ 
We believe that answers must be found to seVeral types of questions: Dot 
, principals hold ne^a t i ve a 1 1 i t udos toward science? What ^iro 'the relationships 
botv/e(!n teachers and students attitudes toward mathematics? Is lack of subject 
matter competancy tlie primary problem in a f^ivcn impact region, or is it low 
fitudent interest? Our evaluation strategy is desi j^ned to c^^tiher data relevant 
to Ihor.r. and many otjflfT questions. If sucli information helps decision makers 
to achieve tl)t: yonlr. of iho Comprrdieris ive I'ro^nm, tlian this evaluation stratej^y 
will liavf' be<'n suerrr^ss fnl . 



/ 
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